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ABSTRACT , , . ^ , 

Delay of feedback studies with animals or with tasks 

which are simple for human subjects have xed to the common assertion 
taat computer-assisted- instruction (CAX) systems should incorporate 
iinmeliate feedback in order to maximize learning. Howev-*r, studies 
using more complex learning tasks have suggested that elays of up to 
at least a day actually benefit learning. A study war esigned to 
investigate the optimal feedback delay foi different „ pes of 
mathematical material in an undergraduate mathematlfrs course taught 
by CAI. The results indicated that immediate feedback* v/as not 
superior to delayed feedback, although the optimal dela'* is very much 
dependent on the type of material involved. The more cc ‘, lex the 
material to be learnr l is, the shorter is the feedback .if. jy which 
can be tolerated without a decrement in retention. Thus^ -"he 
implication of this studi was that time- shared CAI system.^ nued no 
longer be concerned about the resulting delays in input-oui^ ui: 
processes. (RH) 
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Feedback Interval has been defined as the length of time inter- 
vening between the subject's response to a learning or test time and 
the occurrence of an Informative event that tells him whether his 
response is right, wrong, or somewhere In between (Brackblll, Wagner, 
and Wllscn, 1964). One of the major Issues In feedback research is 
now, and has been for quite some time, the question of the ide.'*l / 

Interval between response and feedback. This problL^m has been thoroughly 
investigated la the field of animal learning; considerable research has 
shown con-;lu 53 .'.- .ily that for sub-human organismi- duration ct" feedback 

*'U- 

delay is directly related to decrease in learning efficiency (Rennner, 

1964). With delays of but a few seconds, learning muy not occur at 
all» Studies reporting the superiority of immediate knowledge or 

I 

results over delayed feedback for human subjects have also appeared with 
some regularity during the past three decades. This research, however, 
has been concerned primarily with non-meaning £ul verbal materials. In 
the last few years, a number of studies have cast some doubt upon the 
principle of immediate knowledge of results fcr certain highly important 
types of learning in relatively mature human organisms (English and 
Kinzer, 1966). 

1 
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Rtt« «nd mtn havtt y^aldad surprliingly similar results for a 
variety of simple learning nr.d conditioning situations (Brackbllli 
Wagner, and Wilson. 1964); It is ludicrous, however, to expect parallel 
result!* f.Qt rtMdles involving complex aspects of behavior, 'riie human's 
response to feedback delay Involves just such complexity. Sassenrath 
and Yonga (1968) have questioned the logic of Inferring from a rat 
learning to press a bar or a pigeon learning to peck at a disk, to a 
child learning to speak his native language, or an adult learning matrix 
algebra. The implication is that the principle of reinforcement may 
not have the generality :*.t has commonly been assumed to have. In addi-> 
tion, they point out that in animal studies of delayed reinforcement 
the animal has to rememlier his response over the delay interval and is 
then given only the reinforcement without being again presented with 
the original task, with alternatives. In studies with humans, the sub- 
ject may also have to remember his response over the delay period but 
when receiving feedback he is usually presented with the question or 
task as well as the alternatives originally available. 

Evidence from studies on human motor skills first called into 
question the superiority of immediate feedback on learning; approxi- 
mately eighty percent showed luv s.;.gnif leant differences in learning 
efficiency (Sassenrath and \\-.i;£e, 1968). Further, classroom applica- 
bility was limited by at least two factors. First, the majority of 
these studies im'olved either acquisition criteria or immediate reten- 
tion criteria (More, 1969). The value of such criteria is limited 
because long-term rather than shet t-t-irm retention is generally a pri- 
mary instructional objective. Secondly, th*’ resuits of many studies 
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which did utilise long-term retention criterie were confounded by the 
amount of feedback given; for example > groups which received immediate 
feedback also received delayed feedback in the form of discussion along 
with the delayed feedback groups. 

In recent studies, the principle of immediacy of feedback has 
been further opened to question (Brackbill, Wagner, and Wilson, 1964; 

English and Kinzer, 1966; Sassenrath and Yonge, 1968). These studies 
have found a greater amount of retention by students for whom feedback 
had been delayed than by students who received immediate feedback. The 
realization has grown that the complete acceptance, without qualifi- 
cation, of delayed reinforcement or information feedback generaliza- 
tions, which suggest that a few seconds delay between response and rein- 
forcement may mean the difference between maximal learning and no learn- 
ing whatever, is unwarranted in human learning. 

It is also clear, that the ability of a teaching machine to give 
immediate knowledge of results may not be a valid reason for preferring V 

automated instruction to traditional instruction. The above discussion 
has major implications for computer-assisted instruction (CAI) . Under 
wAI instruction, the student finds out immediately upon responding 

r 

whether his response was correct (in contrast to traditional instruc- 

I 

tion which typically Involves delays of one or more days). The pro- 
vision of immediate feedback has long been presented as a rationale 
for the use of CAI. 

The most obvious question that arises is why such an erroneous 
conclusion has been perpetuated through many hundreds of teaching 
machine studies. One reason is that virtually all of the research 
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hfii Involved Iromedlate retention of non-meenlngful materials (Brackbill 
end Kappy, 1962). Aa firackblll and Kappy have pointed out, although 
moat psychological variables that affect both learning and retention 
have directionally correlated effects on the two processes, there are 
exceptions, e.g., partial reinforcement. Also, at least part of the 
answer lies in the fact that the teaching machine makes simultaneous 
use of several learning conditions (e.g., active rather than passive 
responding) . Whenever two or more variables are allowed to operate 
simultaneously in any situation, the outcome of that situation cannot 
be pinpointed to the influence of any one of those variables because 
their separate contributions to the outcome are inextricably confounded 
(Brackbill, Wagner, and Wilson, 1964). Obviously, the way to increase 
the efficiency of CAI - or any other complex learning situation - is to 
systematically study the separate contribution of each of the identifi- 
able variables. Only by weeding out the ineffective variables, such 
as immediate feedback, can the complex be Improved. 

Review of the Literature 

The literature on delay of feedback contains investigation of two 
dimensions of the phenomenon: feedback received during a learning situ- 

ation (acquisition feedback) and feedback received following a testing 
situation (test feedback). Acquisition feedback studies have generally 
utilized discrimination and paired-associate learning tasks. Test feed- 
back studies have typically involved meaningful material and multiple 
choice evaluation techniques. 

Acquisition Feedback . Most of the research on this dimension 
has been conducted by Brackbill and her associates with feedback delays 
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up to 30 seconds. As their studies have involved discrimination and 
palred-*a 880 ciate learning tasks » the ind .^x of retention has typically 
been relearning scores, or trials to reach criterion. Tirie results of 
two studies with third grade males on a drawing discrimination task 
favored delaying feedback. These studies used delays of 0.5 and 10.0 
seconds (Brackblll and Kappy, 1962) and 0.0, 2.5, 5.0, 10.0, 20.0, and 
30.0 seconds (Brackblll, Adams, and Reaney, 1967). In every case, the 
longer feedback delay improved retention when measured one day after 
original leamingj no consistent results, however, were found when 
retention was measured eight days following original learning. 

Another pair of studies also involved the discrimination behavior 
of third grade males but involved more difficult tasks (Brackblll, Bravos, 
and Starr, 1962; Brackblll, Wagner and Wilson, 1964). Both studies 
used feedback delays of 0.0 and 10.0 seconds. When the task Involved 
drawings, a 10.0 second delay again Improved retention as measured 1 day 
later; for the learning of English-French equivalents, there was no 
difference. In both cases, however, the 10.0 second delay resulted 
in greater delayed retention as measured seven or eight days following 
original learning. 

In 1966, in a series of studies using undergraduate males, Lintz 
and Brackblll found the delay of feedback effect to be dependent on the 
nature of the task. Using a 0.0 or 10.0 second delay in a discrimina- 
tion task, there were no retention differences either 1 or 7 days after 
original learning; for a paired-associate task, however, a 10.0 second 
delay resulted in greater retention both 1 day and 7 days after original 
learning. 
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Based on the above findings, the following conclusions seem war- 
ranted for acquisition feedback: 

1. For a task of medium difficulty, the short-term retention of 
young males is improved by feedback delays of up to 30 seconds; 

2. For a task of greater difficulty, the long-term retention of 
young males is Improved by feedback delays of at least 10.0 seconds; 

3. For a discrimination task, the retention of male under- 
graduates is unaffected by a 10.0 second feedback delay; and 

A. For a paired-associate task, the retention of undergraduate 
males is Improved by a 10.0 second delay. 

Test Feedback . The results are more consistent for multiple 
choice testing situations. Typically in such studies, the student 
responds to multiple choice questions receiving feedback either immedi- 
ately after responding or after a pre-determined time Interval has 
elapsed. Reported feedback delays range from 1 hour to 1 week (1 hour, 

2.5 hours, 1 day, 2 days, 4 days, 1 week). Wlalle some studies have 
investigated completeness of feedback, this report will be concerned 
only with feedback situations in which the item stem and all alterna- 
tives are re-presented along with the indication of the correct alterna- 
tive. Also, as the current study was concerned primarily with delayed 
retention, the results of previous research will be discussed with respect 
to their delayed retention flAdings^!. 

With the exception of one study (More, 1969), the subjects for 
the studies of delayed testing-ieedback have been college under- 
graduates. The subject matter involved has spanned a range of curriculum 
areas from introductory psychology to statistics, and retention has been 
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-measured either three days, five days, or one week following original 
learning. Xn every case. Immediate feedback has been found to be 
Inferior to feedback delayed for 1 hour, 2.5 hours, 1 day or 2 days. 
Prolonged feedback delaya of 4 days or 1 week, however, have been found 
to be inferior to delaya of 1 hour, 2.5 hours, 1 day, or 2 days. These 
results suggest the existence of a non-linear relationship between 
feedback delay and retention. 

Given such a relationship, a delay of 2.5 hours, for example, 
would be expected to result In higher retention than a delay of 0.0 
seconds, whereas a delay of 4 days would be expected to result in 
lower retention than a delay of 1 day. In other words, we would 
expect that increasing the feedback delay would correspondingly 
increase retention (up to some ideal delay period), and that further 
delay would begin to decrease retention until a retention level equal 
to that resulting from immediate feedback were reached. Examination 
of the results of previous studies supports this hypothesis: 



Feedback Delay Results 
0.0^1 hour, 2 days ^ 1 v;eek 
0.0^ 2.5 hours , 1 day > 4 days 
0.0 •<. 1 day 



Author (s) 

English & Kinzer (1966) 
More (1969) 

Sassenrath & Yonge (1968) 
Sturges (1969) 



0.0 1 day 

It appears that delayed testing-feedback actually improves retenttv a of 
the learned material while both Immediate feedback and overly-prolonged 



feedback delays impair retention. 
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The purpose of the present study was 1) to systematically Inves- 
tigate the feedback delay variable » and 2) to extend the research on 
feedback delay to mathematical learning. In a CAl learning task, three 
types of mathematical material were investigated: definitions, algo- 

rithms and proofs. The four feedback conditions were as follows: 

1. Immediate feedback (IF) - students received feedback immedia- 
tely following a with in-pro gram response (WPR) ; 

2. 30 second feedback (30S) - students received feedback 30 
seconds after making a WFR; 

3. End of session feedback (ES) - students received feedback 
on all WPR's at the end of each day^s sessioA; and 

4. 24 hour feedback (24H) - students received feedback on all 
WPR's made during one session 24 hours later at rhe beginning of the 
next session. 

Based on previous findings with college students, the following 
hypotheses were formulated: 

1. For definitions and algorithms, 30S, ES, and 2411 feedback 
will result in greater retention than IF. Also, the means for stud<=-nts 
in the 24H condition will be systematically higher than the means for 
students in the ES and 30S feedback conditions; similarly, the means 
for the ES condition will be systematically higher than the means for 
the 30S condition. 

2. Due to the sequential dependencies involved in a mathematical 
proof, the IF and 30S conditions will result in greater retention than 
the ES and 24H conditions. 
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METHOD 

Subjects 

Subjects for this study were lower-division undergraduate volun- 
teers from tw' institutions: Florida State University (n ■ 40), and 

Tallahassee Community College Cn * 40). The subjects came from a 
variety of curriculum areas and all received an honorarium for their 
participation in the study. 



Learning Materials 

This study utilized portions of CAI materials previously described 
in detail (Love, 1969), which were adapted for the present study. The 
Instructional program was composed of approximately four hours of instruc 
tion (20 lessons) on Boolean algebra, primarily set theory. The material 
Included definitions, algorithms, and proofs. The content of the les- 
sons was as follows: 



Lesson 


1 


Lesson 


2 


Lesson 


3 


Lesson 


4 


Lesson 


5 


Lesson 


6 


Lesson 


7 


Lesson 


8 


Lesson 


9 


Lesson 


10 


Lesson 


11 


Lesson 


12 


Lesson 


13 


Lesson 


14 


Lesson 


15 


Lesson 


16 


Lesson 


17 


Lesson 


18 


Lesson 


19 


Lesson 


20 



Introduction to Set Theory 
Venn Diagrams With Two Sets 
Venn Diagrams With Three Sets 
Special Cases 
The NOT Operation 

Determining A Set Given Any Expression 

Boolean Algebra Introduction 

The Commutative Laws 

The Associative Laws 

The AND Distributive Law 

The ^OR Distributive Law 

The Universal Element 

The Null Element 

Complements 

Proof of Theorem - Universal ^ 

Proof of Theorem - Null AND 
Proof of Theorem - ^ Indempotent 
P'roof of Theorem - AND Indempotent 
Proof of Theorem - ^ Absorption 
Proof of Theorem - OR Identity 
(See Appendix A for a Content Outline) 



Within-program problems were presented in a multiple choice format. 
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Apparatus 



The learning materials were presented by an IBM 1500 CAI system. 
Terminals for the system consist of a cathode-ray tube, a light pen, 
and a keyboard. The terminals were located In an alr-conditloned, 
sound-deadened room. The CAI system administered the Instruction and 
recorded the students' responses. 

Experimental Design 

All students in all groups were pretested, received their Instruc- 
tion on days 1-4, and were tested for delayed retention fourteen days 
from the first day of original learning (Sec Figure 1). All students 
received the same instruction and the same wlthln-program testing 
frames; the. groups differed only as to when they received their feed- 
back for wlthln-program testing frames and with respect to occurrence 
of quizzes. 

Experimental Procedures 

This study was conducted over a six-moo.th period. Each week 
between 12 and 16 students were processed from one of the 2 participating 
institutions. These students were randomly assigned to one of the 
four feedback conditions. Of the forty students from each institution, 

10 were assigned to the IF condition, 10 were assigned to the 30S condi- 
tion, 10 to the ES condition, and 10 to the 24H condition. For all 
students, the procedure was the same: 

1. At a time convenient to the student, he was administered a 
100-item multiple choice pretest convering all aspects of the instruc- 
tional program; the pretest was a re-ordered version of the delayed 
retention (DR) test; 
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Figure 1 . —Experimental Design for the Four Feedback Conditions 



2. On thtt first day of Instruction, the student worked through 
lessons 1-5 J each lesson consisted of Instruction, illustrative dia- 
grams, and multiple choice "r.st questions; 

On the second day of Instruction, the student worked through 

lesBotui 

A. On the third day of inpferuction, the student worked through 
lessons 11-lA; 

5. On the fourth day of instruction, 5:he student worked through 
lessons 13-20; 

6. On the fifteenth day, the stude^ i* was administered a 100- 
item DR test. 

During the instruction, students ir* the IF group received their 
feedback immediately follow’ r>g aach WPR; the feedback wa.^’ of the form 

I 

"correct, the answer is ” . or, "incorrect, the answer is * 

They w,ere administered a quiz a* the conclusion of eacn daily sc'’ s*. ’.i» 
Each quiz was a •. .iltcsmate form of a portion of the DR test; th»* four 
quizzes had a total of 100 items. Students in the 30S group receive vi 
feedback 30 seconds after making a WPR and were administered a quiz at 
the conclusion of each session. Students in the ES group did not 
receive feedback on their W?R*s until the end of the session; at that 
time each question and its alternatives were presented. For each 
qu.^fc-}tion, the WPR that the student had chosen was indicated, At the 
conclusion of the feedback phase, the student was administered the 
quiz for that session. Students in the 2AH group also did not receive 
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CetdbAck on their \^R*i during the lnetructlon.« The following deyi at 
the beginning of the next aeeiion» they received feedback for the pre- 
vioui day 'a leaaona ae deecribed for the ES group. Following feedback» 
they were administered the quis for the previous day's lessons. 

The pretest, all quisles, and DR test were administered 
via paper and pencil; they were not part of the CAI instruction. 

RESULTS 

Re liability 

Reliability was computed for each institution to insure that 
the criterion test was equally reliable for the two student popular 
tions. The criterion test consisted of one item for each concept 
included in the learning materials and therefore contained three sub- 
strata corresponding to the three types of mathematical learning 
investigated, tuitions, algorithms and proofs* The KR-20 reli- 
ability was computed for each sub-stratum as well as for the total 
test (See Table I). 

TABLE I 

Reliability of Criterion Test, by Type of Material 



and for Total Test 



Institution 


Reliability 


Definitions^ 


Algorithms^ 


Proofs^- 


Total 


Florida State 
University 


.87 


.90 


.96 


.96 


Tallahassee 
Junior College 


.81 


.88 


.93 


.95 



^Number of items - 23 

^Number of items = 50 
'^Number of items = 27 
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Ai Tabltt 1 indicates I raliabllltiaa wara aqually high for both 
•tudant populationa. Conaaquantlyi data for tha two groups wara 
combined for analysis. 

Retention 

The two factors of interest were feedback delay and type of 

material. The Ideal analysis design would be a 4 x 3 analysis of 

covariance with repeated measures on the second far .:or (covarying 

for pretest differences). However, the scores for the three types 

of material were not based on the same number of items, there were 

23 definitional Items, 50 algorithm items and 27 proof items. In 

Order to equate the scores, it would be necessary to either covert 

them to proportions or to apply a transformation. Neither of these 

approaches Is desirable, as they affect both the reliability and 

the generalizability of outcomes. 

As no acceptable analysis procedure was available, the adjusted 
3 

means for each group, by type of material, were computed and con- '"d 

verted to percents (See Figure 2). While no probability levels 
can be attached to such statements, the following observations 
are offered: 

r 

1. There appears to be an Interaction between feedback 
delay Interval and type of material. 

2. Up to 12% more definitions were retained by students 
In the 30S group, than by students In the other three 
conditions. 

3. Students In the 24H group retained up to 23% fewer 
proof Items than students In the other three conditions. 

4. Students in each of the four feedback groups retained 
algorithms to approximately the same degree. 
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Percent Retention on Delayed Criterion Test 
(Based on Means Adjusted for Pretest Differences) 



I 



IS 




Feedback Condition 
(n=20 for each group) 



Fig, 2. — Percent retention for each feedback condition by type 
of material. 
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DlSCUfiSION 



Past research on delay of feedback has been exclusively involved 
with a variety of verbal learning tasks. This study Investigated the 
phenomenon for three types of mathematical learning, definitions, 
algorithms and proofs. Results Indicate that the optimal feedback 
delay Is very much dependent on the nature of the learning material, 
although they do support previous findings with regard to the non- 
superiority of Immediate feedback. 

Verbal learning research has generally found a one-day feed- 
back delay to be optimal and delays of four or more days to be harm- 
ful; this study on mathematical learning has found that while a 
delay of thirty seconds may be beneficial, a one-day delay is equally 
effective at best, and may have a seriously debilitating effect on 
retention. In the present study, the definitional material was most 
like the learning material in past research and it was the only type 
of material for which a delay of feedback appeared to facilitate 
retention. The instruction on proofs contained the material most 
unlike that utilized in past research; correspondingly a one day 
feedback delay adversely affected retention of this material by as 
much as 23%. In conclusion, it appears that the more complex the 
material is, the shorter is the feedback delay which is defined 
as being ''overly-prolonged.*' 

The implication of this study for computer-assisted mathe- 
matical instruction is that immediate feedback is neither necessary 
nor desirable; a slight delay may even facilitate retention. One of 
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tht problems generally associated with simultaneous operation of 
terminals by many users » Is the accompanying delay In Input-output 
processes. This study supports previous research In the contention 
that this need not be a consideration. 
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APPENDIX A 

CONTENT OUTLINE OF CAI INSTRUCTION 

Lesson 1 - Introduction to Set Theory 
Definition of a Set 
Complements 
Universal Element 
Null Element 

Lesson 2 - Venn Diagrams With Two Sets 
The AND Operation on Sets 
The 2R Operation on Sets 
Lesson 3 - Venn Diagrams With Thic:e Sets 
The MD Operation 
The ^ Operation 
Lesson 4 - Special Cases 
Combining AND and 0 
Combining OK and 0 
Combining A^ and I 
Combining ^ and I 

Combining MD and Self, e.g. A • A - A 
Combining OK and Self, e.g. A + A * A 
The AND Complement 
The ^ Complement 
Lesson 5 “ The Not Operation 

Lesson 6 - Determining a Set Given Any Expression 
Equivalent Expressions - DeMorgan’s Laws 

o 

ERIC 
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Lesson 7 Boolean Algebra Introdutlon 
Variables 
The Universal, I 
The Null, 0 
AND . OR, and NOT 

Introduction to the Laws of Boolean Algebra 

The ^ Commutative Law 
The AND Commutative Law 
The OR Associative Law 
The AND Associative Law 
The ^ Distributive Law 
The AND Distributive Law 
The Universal Property 
The Null Property 
The AND Complement 
The ^ Complement 

Lesson 8 - The Commutative Laws 

The OR Commutative Law 

The AND Commutative Law 

Lesson 9 - The Associative Laws 

The ^ Associative Law 

The AND Associative Law 

Application of the Commutative and Associative Laws 
Lesson 10 - The AND Distributive Law 
Lesson 11 - The ^ Distritutive Law 
Lesson 12 - The Universal Element 
The Universal Element 
The Universal Property 
Lesson 13 - The Null Element 
The Null Element 
The Null Property 
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Lesson 14 - Complements 
The OR Complement 
The AND Complement 

Lesson 15 - Proof of Theorem 1 
Universal ^ 

Lesson 16 Proof of Theorem 2 
Null MD 

Lesson 17 - Proof of Theorem 3 
OR Idempotent 

Lesson 18 - Proof of Theorem 4 
AND Idempotent 

Lesson 19 - Proof of Theorem 5 
OR Absorption 

Lesson 20 - Proof of Theorem 6 
OR Identity 



FOOTNOTES 
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